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MANUAL ENTRY INTERACTIVE PAPER AND 
ELECTRONIC DOCUMENT HANDLING AND PROCESSING SYSTEM 

FIELD OF THE INVENTION 

The invention relates to the field of handling paper documents and 

5 their corresponding electronic versions by allowing the user to interact with 

both versions of the same document simultaneously. 

BACKGROUND OF THE INVENTION 

Because people prefer paper documents for some tasks such as 

reading or annotating documents while electronic documents seem to be 

10 preferred for tasks such as storing, searching or communicating, it is 

desirable to provide document systems that maximize the user-friendty 

aspects of both approaches. 

Unfonunately. there has been a large technological barrier between 

the paper and electronic documentation worlds. For example, electronic 

15 symbols may be readily convened into a printed page, but printed 

documents are not readily translated into compact electronic symbols. 

Also, the human interface with each of the mediums are radically different: 

writing on paper (or similar surfaces) comes naturally to children, while 

creating electronic documents generally requires the use of application 

20 software, keyboard, mouse, display, etc. 

The present invention provides a homanrinterface input/output 

device that is: 

(a) small and portable. 

(b) natural to use as a pen. 
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(c) works with both electronic and paper documents, and 

(d) avoids complications and inconveniences ol devices such as 
bulky scanners and digital pads. 

As a result, users can readily create or manipulate paper documents 
5 while simultaneously making corresponding electronically-coded Images of 
the same created or manipulated paper document. This is accomplished 
by means of a hand-held, pen-like instrument that can write on paper 
documents, simultaneously sense the writing, and interpret and enter the 
written markings in the electronically-coded document. Also, image 
10 sensing allows the system to identify the type of document being subjected 
to manually-written modifications. 



SLiMMARY OF THF fMupf^j./^,., 

A manual entry interactive handling system uses a writing instrument 
to manually write data on an ordinary document (that can have pre-printed 
information) and automatically interpret the wrinen data so that a 
corresponding electronic document containing any pre-printed document 
information is modified by adding the written data to the stored electronic 
document. Both pre-printed data and the interpreted written data can be 
electronically displayed, the system comprises: 

(a) a physical document page with a surface suitable to be written 
upon that can also contain pre-printed document information; 

(b) a pen-instrument for writing on the physical document surface, 
the pen-instrument having suitable transducers for sensing 
both the written data and the multi-dimensional physical 
forces (e.g. up to three rectilinear and two rotational 
components) exened on the pen instrument when writing and 
for reading pre-prinied digital codes, and a communication 
link to transmit the transducer data to a base unit; and 

(c) a base unit lor receiving the transducer data and converting 
the transducer data to binary digital information suitable for 
processing in the base unit CPU. or another computer, for 
document identification and for incorporating the written 
information with the pre-printed document information. 



In another embodiment, the transducers are capable of defining pen 
motion resulting from movement of the pen-instrument for other than writing 
such as pointing for the control of an object (such as lights). 



BRIEF nP«;pq.p^^r^^| T HE DRAWf| ^r^<; 

The present invention is iliustraied by way of example and not 
'imitation in the figures of the accompanying drawings, in which like 
references indicate similar elements and in which: 
Figure 1 shows the formal of a daily CB page. 
Figure 2 is a flow diagram of the basic operation of a CB system. 
Figure 3 is a flow diagram for procedure 600 for interpreting the 
pen-instrument input data. 

Figure 4 is a flow diagram of the entry select procedure 700. 
Figure 5 shows a CB system's major components. 
Figure 6 shows the architecture of a pen-instrument. 
Figure 7 shows the geometry of an infrared PI positioning system. 
Figure 8 shows an arrangement of strain gauge transducers in a 
pen-instrument. 

Figure 9 shows the interconnections of the strain gauges on the 
flexible diaphragm. 

Figure 10 is a triple bridge circuit for processing the strain gauge 
transducer data and for producing x. y. and z force components. 
Figure 1 1 shows a PI unit block diagram. 
Figure 12 shows a CB base unit block diagram. 



DETAILFO OESCRIPTIOM 
A- Application Ar^^^ 

An Interactive document system based on a pen-like instrument, or 
pen-instrument lor convenience, has application in numerous document 
systems that include, but are not limited to. the following systems, 

Cg'gndar Book: appointments entered on a physical page of a 
calendar book are sensed, interpreted, and 
recorded on an electronic calendar as well. 
Fprms Prpof>fging: filling out of a prescribed form wouW 

automatically cause the generation of a 
corresponding electronic form, the paper version 
being kept, as needed, for legal or safety 
reasons. A significant cost reduction in 
computer data entry results. 
Signgiwrg Vgrifig^tipn: improved reliability of signature recognition 

obtained by storing the signature related 
time-course by using spatial location and 
pressure to characterize the signature. These 
parameters are not available to the forger from 
viewing the signature. 
*ni?r9<;:five Fax: a facsimile system that permits users to receive 
documents by means of "links" similar to 
WebFax. This woukJ permit the user to fonvard 
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pages to selected users by simply writing the 
desired destination, which when sensed and 
interpreted would provide the communication 
"link-. Also, the user can annotate the document 
and the transmit or store the document with the 
annotation without the use of a facsimile 
scanner. 

Bggvmgnt F^itinq : any paper document can be edited by 
crossing-out words, and by the use of 
typographic notation lor other editing operations, 
or annotated, or sent elsewhere. The paper 
modifications immediately affect the electronic 
version so that the electronic version can serve 
as the true source document because any 

15 

access of. or communications of. that document 
would rellea all edits or annotations. 
Infprmf^tion Mj^npnomp n i : scanning the description of a reference 

document can have that paper (or its reference) 
stored in a designated file without photocopying, 
ggg vment Retrieval : searching of electronic databases by writ.ng or 
circling -a few keywords, thumbnail descriptions, 
or by drawing pictures, which, when interpreted, 
provide the data location. 
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Note Book : using the pen-instrument tor writing notes on 
paper and for electronically retrieving notes 
previously written on paper and stored from a 
sequenlially ordered (and/or an otherwise 
organized) elecironic memory file. 

B. Svslem Details 

B.I r.alPfiriar Book (CB) 

Because of the variety of possible document systems, as outlined in 
the preceding section, the details of the invention will be presented using a 
single application from which the extension to other applications will be 
readily understood by those practicing the art. For this reason, the 
Calendar Book is described below. 

The Calendar Book , or -CB" hereinafter, looks like a conventional 
calendar having a page representing a segment of calendar time: a day. a 
week, a month, or a year. Each segment of time is further subdivided as 
needed: a year into months and weeks, a month into weeks and days, a 
day into hours and fractions thereof. Digitally coded information is printed 
on each page so that the pen-instrument can be used to read the pertinent 
digital code that identifies the page format and the segments of calendar 
time spanned- by the.CB page. 

Figure 1 shows a page 20 from a CB using a one day per page 
format. This version allows selectable information options to be printed for 
each day. An area is set aside for a bar code identifier 13 for specifying the 



date (year, month, and day) and user I.D. The date (year optional) is also 
printed in area 21. Column of rectangles 22 and 22* are available for 
selecting and identifying the time by entering a suitable mark (such as a 
check mark) in the appropriate rectangle. Column 22 and 22* have 
identical meanings and are provided so that two entries, only 15 minutes 
apart may be entered each using a half width and indicated as separate by 
dividing line 23. For example, an appointment with Baker is indicated for 
10:00 a.m. while an appointment with Charlie is indicated for 10:15 a.m. In 
other words, each hour is divided into 15 minute-segments in this example. 
Other increments such as one-half hour and 10 minutes could also be 
specified. The user's choice would only be constrained by the application 
being served and the page size (or detail) available. The time of day is 
also indicated by the position of a written entry for any given calendar page 
format. The pen-instrument provides the writing point and the optical, the 
positional, and the writing point to paper pressure data to the computer for 
interpretation, storage and graphical display. 

The graphical display can provide immediate feedback by use of an 
optional cathode ray tube (CRT) display. The feedback can be a simple 
recreation of the written message by using pen-instrument positional 
writing point pressure data or the interpreted written message from a 
cursive character recognition device using the same data. In the latter 
case, incorrectly recognized written data can be scratched-out or 
over-written. 



t 
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Bar codes are read by a bar code scanner on the side of the 
pen-instrumenl. The bar code recognition device may either constantly 
scan in search of a legitimate bar code or operate when the positional 
sensors indicate that the pen-instrument is in a horizontal position suitable 
5 for reading a bar code. 
B.2 Operating Mode 

Figure 2 is a llow diagram describing the basic operating mode of a 
particular CB. The operation begins at step 500 where ci is assumed that a 
fixed format has been prescribed and its description is stored with the 

10 configuration information for that CB. Configuration data includes calendar 
type, chronology layout defining time intervals, segments, and writing 
spaces. Also, a user list is stored against which the bar-code I.D. can be 
checked for validity. 

At step 501 , the computer checks if the pen-instrument is moving 

1 5 horizontally, indicating that a CB operation is about to be initiated by 

reading the bar-code located in area 1 3. If not. the system goes into a wait 
loop in which the pen-instrument motion is monitored. If the appropriate 
horizontal motion is detected, the process proceeds to step 502 in which 
the bar-code I.D. is read by scanning bar-code area 13 with the 

20 pen-instrumenl. If a valid bar code signal results, step 503 moves the 
process to step 505. otherwise the process goes to step 504 where an 
audible and/or visual alarm is set (or informing the user that access is 
denied and the system returns to step 501 . The bar-code can also be 
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co-npa^a wim a fe, o. aumo..ec, coaas ana « a „,a,ch ax,s,s ..e CB 
=cc«s,aq„es„svatoe<.ancp™ce«.s,os,ap505. A„Nsjuna„,a ,he 
pe.-.ns,n.mem usa, .a. ,y ,o read .he .a,^a a,a.„ o. .nvastigaia ihe 
syaem (or Ihs cause of ihe misread bar-code data. 
5 A, s,ep 505. ,he eleclronic version o, ,he CB paje can be dispiaved 

on a screen (CRT or UCO, .^n ,ne dispia. inCudinp a„ prior en.es on ,na, 
page. The process enters anolher „ai, loop Ihrough ,es, s,ep 506 which 
cecs „ ,he wrtlna poi„, pressure signal exceeds .hreshoid „ i, does ,his 
« .aKen as an indication ,ha, a pen-instrumen, operation is initiated and the 
P'ocess moves to step 5„ „nere the pre.ous »ait toop index ,s,ep 507, is 
c eared. Othe™.se. the process goes into a »ait toop compris,ng step 507 
2 -h.s the amount o, parses (ortime, that the ioop is traversed. Step 
50a tests i, a preset time has been exceeded, and i, not goes bac* to step 
S06^ Othenvise. the toop count index o, step 507 is Ceared in step 509 and 
.he dispia, is aboned in step 5.0 betore retunting to step 50, , 

Step 5,2 es.ablishes .he ,oca.ion o. the whtten enhy by using the 
gx'oscopic rate i„,o,mat,on. Sy Knowing the tocation o, the bar-code page 
i«-.i«ca.ion and by in.egra.,n9 gyroscopic velocity inlormation the 
location ,-. „ 

inlormation is used ,o 'o^'^i'^^^.lZZ.l^cuZ^^T 
electronic graphical display. 

From s.ep 512. ,he process moves ,o step 600. a procedure for 
-nierpretir^g „e message ,ha, the pen-instrumer,t has entered on the CB 
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page. The procedure is shown in greater deiail in Figure 3. The 
muKi-sensor output signals from the pen-instrument are in electrical form, 
preferably as binary coded data. In this example, four data types are used 
and processed: pen-instrument pressure processed in processor step 601 , 

5 gyroscopic restoring forces in processor step 602, optical data 

representative of the pen-instrument written message in processor step 
603, and an auxiliary data signal representative of special cues for 
identifying calendar and position data from4he printed CB page in 
processor step 604. Each processor step 601 -604 extracts signal features 

10 from its associated data source. A particular written data entry may include 
a number of distinct message unit such as a time code derived from the 
printed CB page, and a written portion that may include a name and place. 
The message input sequence is parsed into message segments in vector 
processor step 605 using the various processed sensor data from 

1 5 processors 601-604. Thus, the presence o1 the writing point-to-paper 
pressure over an interval of time indicates writing a message segment. 
Gaps between pressure intervals may indicate the gap between segments. 
A gap can also indicate a crossing of a "t" or the dotting of an "i" after the 
word is written. In the latter case, gyroscopic data can confirm the 

20 backward pen-instrument: moirnn-and subs^i iftnt pen-instrument pressure 
coupled with either a stationary dotting action or a fonvard sweeping 
crossing of a T By using time-synchronized multi-sensor data, the 
pen-instrument written message can be used to parse the input message 
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into dislinct and interpretable acts. Parsing vector processor 605 produces 
a set of vectors describing each written segment which are then identified 
by comparing the parsed vectors with a stored set of exemplar in exemplar 
store unit 607 by comparator step 606. In step 608. each parsed segment 
generates a set of candidate vectors corresponding ,o a plausible parsed 
message segment, which are preferably ordered in descending order of 
plausibility, i.e.. of being the correct message segment. 

The plausible candidate message units from message selection step 
608 are arranged by sequence index. 0 < k < K-1. indicating the order of 
occurrence of K message segments. The k>h message unit has U 
plausible candidates designated by .ndex I,, where 0 ^ Ik s U-l and 1^ 
indicates decreasing plausibility with increasing value of Ik. 

Because the system should provide a means for displaying a 
•acsimile reconstruction of the written message, the output of parsing vector 
processor 605 also provides an output to locus reconstruction generator 
609 that operates on the gyroscopic and pen pressure data to reconstruct 
the locus Of the pen point while writing. The digitized locus is passed on to 
entry point E of Figure 4 for conflict resolution. In some applications, a 
facsimile of the original written message is preferred and in others it 
provides a back-up to an alphanumeric interpreted representation based 
on character pattern recognition techniques. 

Step 700 is a multi-step process as shown in Figure 4. At step 701 . 
the message unit index, k. and the plausible candidate index. Ik. are 
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Cleared {k=0, lk=0). Al step 702 the Ik candidate of the message unit is 
retrieved from memor/ and displayed graphically as trial message unit 
super-imposed on the appropriate CB page at the location corresponding 
to that written on the actual page calendar by the pen-instrument in step 
5 703. At this point in time, the CB user is allowed to examine the trial 

message unit in test step 704. Three exits from step 704 are provided: a 
wait loop (similar to that previously described) comprising wait loop 
indexing step 705 and test step 706 that examines the wait loop index to 
determine If the process shoukJ be aborted by returning to input B of Figure 

10 2 or returning to step 704. If the user indicates by the pen-instrument that 
the candidate trial message unit is correct by making an appropriate 
pen-instrument movement (such as simulating or actually writing a check 
mark if correct and an "X" make if not), the process moves to step 710 
where the wait loop index is reset and then step 71 1 where index k is 

1 5 incremented and Ik is set to zero. At step 712 a test is made to determine if 
all message units have been displayed and, if not, the process returns to 
step 702. 

If the user indicates al test step 704 that the trial candidate message 
unit is not correct, the process goes to step 707 where the wail loop index is 
20 cleared and then to step 7©87wteie4he^^pfeus^e*:^cnM is 

incremented. At step 709. Ik is checked to determine if all of the candidates 
have been exhausted and. if not. the process returns to step 702. If all 
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candidates have been exhausted, the process returns to the beginning via 
input B of Figure 2. 

If all message units are accounted for in step 712 (implying the 
graphical message on the screen is correct), the process checks to see If 
5 any conflict exists because two events are scheduled for the same time. If 
no conflict exists, step 71 4 stores the CB entry and the process proceeds to 
input B of Figure 3. If a conflict exists, a conflict alarm is given on the 
screen display or by any other means such as an audible alarm (step 715) 
and the user is given time to respond by the wait loop (steps 717, 718). If 

10 the response is not timely, the process aborts by returning to input B of 

Figure 2. If the user indicates that the new message should not be entered, 
step 716 aborts the process by returning to input B of Figure 2. If the user 
indicates at step 716 that the new entry should be entered, step 71 9 asks if 
the previous message is to be deleted and the new entry entered. If the 

15 user indicates by an appropriate signal from the pen-instrument that 
previous message is to be overridden, the process goes to step 720 
otherwise both forms of the written message are retained in step 721. 

Process 700 of Figure 4 is designed to select correctly identified 
candidates from a written character recognition device in step 704. 

20 However, an optional embodiment would provide for sending the 

processed data outputs from processing steps 601-604 of Figure 3 to an 
optional write simulation step 609 where the processed sensor data is used 
to generate a replica of the user's written message without any writing 
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recognition processing steps and is then displayed as a copy of the hand 
written message on the document. Also, both forms of v^ritten messages 
(an ASCII encoded interpretation of the message and a replica of the 
original) can be retained for future reference and selectively displayed on 
i the CB display. By providing this option, the writing recognition process 
does not have to be 100% accurate. 

In this manner, the electronically stored version of the C8 can served 
as the true source document for which the paper version and trail exists in 
Close accord with the electronic version, if the paper version begins to 
become unmanageable because of numerous changes, the user can 
obtain a new .print-out of the page which may be a new page without any 
entries or a new page with all up-to-date entries. 

Because the source exists in electronic form, electronic agents can 
use this information to aid the user by. for example, reminding users of 
appointments, performing group scheduling with other agents, reviewing 
the series schedule, and providing support information lor a given 
appointment. 
^•3 The CR .^iynf 

The CB system 90. shown in Figure 5. includes four major hardware 
components: 

(1 ) a CB page 10 or 20. as previously described. 

(2) a pen-instrument (PI) 91 for provding a user with a means for 
entering data (both printed digital and written). 
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(3) a base-unit 92 for electronically processing and storing data 
received from the PI and including processing and conditioning electronics 
96 and optional graphic screen display 95: and 

(4) a communication channel 93 between the PI and base-unit. 
B.3.1. Pen-lngin/r nent fPn Qi 

PI 91 is configured as a -normal- pen. i.e. a writing instrument, thai is 
to be conveniently held and used for miting on a CB page 10 as shown in 
Figure 6. In addition to providing a writing point that deposits ink (or other 
writing medium) on to paper surface by contact, the PI incorporates a 
battery power source 250. and transducers for providing information for the 
interpretation of the user written data or printed digital information from the 
■ DB page, including the following. 

. Ppsition Sensor piQ : an inenial reference system is provided 
by a gyroscope (gyro) in which a rotating member will continue lo turn 
about a fixed spin axis as long as no forces are exerted on the member and 
the member is not accelerated. Gyro attitude transducers (free gyros) are 
used as two-degree-of-freedom gyros that can provide an output for each of 
two of a PI S three attitude planes: pilch, yaw. and roll (or x. y. and z axes). 
Rate gyros are attitude-rate transducers that provide an output signal 
proportioned to angular velocity (time rate of change of attitude). The rate 
of change of the two axes representing the plane of the CB page are used 
to interpret the motion and relative position associated the use of the PI. 
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Accelerometers may also be used for position sensors by double 
integration of the acceleration signals. 

When the PI is equipped with an infrared (IR) detector, a multiplicity 
of two or more external IR radiating scanning sources located at prescribed 
positions can be used to track the position of the PI by using triangulation 
techniques that measure the angle at which each IR scanning source 
illuminates the PI. Alternatively, the PI can be equipped with an IR 
transmitter and two or more external scanning receivers located at 
prescribed positions can be used to track the position of the PI by 
triangulation. Ftg. 7 shows the geometry of an IR tracking system using 
three external positions labeled 50l'. 5ll. and 521. Each position has a 
directive scanning beam (receiving or transmitting) 502, 512. and 522 
associated respectively with positions 501, 51 1. and 521. The shaded 
area, labeled 530. represents the intersection of the beams and 
corresponds to the estimated position of the PI. 

Presgyrff Spnsprs IQQ: strain gauges (or other pressure 
sensitive transducers such as piezoelectric or reductive force transducers) 
connected to the shaft of the pen tip of the PI are used to record the forces 
applied to the tip by measuring the force exerted on four pairs of orthogonal 
strain gauges mounted on a flexible diaphragm that is attached to the inner 
walls of the Pi with the pen shaft connected to and passing through at right 
angles to the diaphragm center. Figure 8 shows how the pressure sensors 
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are mounted in the Pi. The operation of such a structure has been 
described by Crane et at. in U.S. Patent Number Reg. 29.765. 

Figure 8 shows a PI cylindrical lower end 1 10 that houses a 
ball-point 1 16 on the lower end of shaft 114 containing the ink reservoir 
The Shaft, supported by a non-conducting flexible diaphragm 112 that is 
attached to cylindrical wall 110. causes the diaphragm to fie. when 
pressure is applied to ball-point 1 16. As a result, strain gauges 121-128 
that are affixed to diaphragm 1 12 are strained (or compressed) in 
accordance with the flexing of diaphragm 1 ,2 in response to the forces 
experienced by ball-point 1 1 6. In this manner, the forces experienced by 
ba..-poin, , ,6 are convened into resistance changes in the strain gauges 
121-128. 



Because the strain gauges are paired and mounted on opposite 
s.des Of diagram 1,2. each pair. (121 . 122,. (,23. 124,. (125. 126,. and 
(127. 128,. tends to have opposite forces (tension vs. compression, 
apphed to the upper and lower strain gauges. When a force is applied in 
the x-y Plane (CB page plane, to ball-point 1 16. diametrically opposite 
pa.rs tend to have opposite tension and compression lorces. However 
When a force perpendicular to the x-y ptane is applied, all pairs tend ,0 
expen^ce simi,a.^s,o.tians. Thes..characteris.ics permit the separation 
Of the ball point force to be resolved into three onnogona. components (X 
Y. 2, where X and Y are in the x-y plane and 2 is perpendicular ,0 the x-y ' 
Plane. Thus, by connecting the strain gauges 121-128 as shown in Figure 



20 

9 and applying the strain gauge connections to the multiple bridge circuit 
shown in Figure 10, the ball-point force is resolved at output terminals 1 52. 
174. and 160 into the X. Y. and 2 components, respectively. 

The strain gauge bridge circuitry of Figure 1 1 is preferably mounted 
5 within the PI unit for greater noise immunity. 

Afthough Figure 8 shows shaft 114 parallel to the z' axis and 
perpendicular to the x'-y* plane, the writing surlace of a CB page (x-y plane) 
generally would not be perpendicular lo the axis because a "naturally" 
held pen-instrument would from an included angle between shaft 114 and 
10 the CB page surface of less than 90**. The transformation between two 
coordinate systems (x. y. 2) and (x\ y', 2*) that share a common origin (the 
pen lip) Is readily accomplished by the following vector relationship. 
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where (a.^. v^]. for 1 < n<3, represents the direction cosine vector of 
1 5 the (x. y, 2) unit vector with respect to the (x\ y\ 2') unit vectors (for example 

see Korn and Korn. "Mathematical Handbook for Scientists and Engineers." 

McGraw-Hill Book Co.. New York. 1961. section 3.1-12. "Translation and 

Rotation of Rectangular Cartesian Coordinate System"). 

By knowing the angular orientation ol shaft 1 14 with respect to the 
20 CB page (x-y plane) and the (x*. y*. 2*) components of force, the horizontal 

and venical forces in the (x. y. z) system are readtly obtained. Also, any 



rotation of the PI coordinates with respect to its inertia! (gyro) system may 
also be accommodated by use of the above transformation. 

The angular rotation of shaft 114 would tend to be the same each 
time the PI is used by a given user. This would permit system to "learn" the 
proper value of the direa cosines or, equivalently, the angles of rotation by 
taking samples of vertical forces (z), horizontal plus vertical forces (x. 2). 
and longitudinal plus vertical forces (y. 2) when a given user respectively 
exens a vertical force on the page, writes it horizontal line, and writes a 
longitudinal line. 

The transformation of coordinates for pen tip forces is not essential 
because the information content of a vector is not changed by a linear 
coordinate transformation. However, it may be useful for the normalization 
of forces so that they are independent of how the pen-instrument was held. 
However, un-normalized forces may be useful for signature verification, 
because of the added parameters relating to how the pen-instrument is 
held and used. 

Pressure sensor information is useful for recognizing when a written 
entry is made on the CB page and for parsing the entry into message units 
for easier message recognition. It is also useful for signature verification. 

Qptiggi S^n^jTL^- a ^ai j ^c oa^&^-^emoer (CCD) optical 
scanning transducer is used for reading reflected light variations from a 
scanned portion of the CB page covered by the CCD array. The transducer 
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is used to scan the identifying CB page bar-code. The CCD array can be 
located on the PI in several different areas: 

(a) a linear side nr>ounted array 220 along the length of the PI is 
useful for scanning a large segment of the page; 
5 (b) a CCD array 260 at the tip of the PI pen or to one side of the 

tip is useful for identifying the local context of the document an exact 
location of the PI pen tip by recognizing pre-printed marks; or 

(c) a CCD array 240 at the back (i.e. "eraser") end of the PI 
because it is more intuitively acceptable than a side mounted array. 

10 Regardless of the location, the CCD sensors must be able to read 

digitally encoded and pre-printed information and support at least 
rudimentary optical character recognition (OCR). The particular choice of 
CCD will dictate the associated illumination requirements. 

A communication path 93 is needed for linking PI 91 with base unit 

1 5 92 where most signal processing of the PI sensor occurs. A wired 

communication link 93 can use a small tether connecting the PI to base unit 
92 but a wireless connection using either infrared or radio techniques 
would be preferred. For this purpose transmitter/receiver unit 230 is 
incorporated in PI 91. 

20 Figure 11 is a block diagram of the PI electronic system showing 

physical inputs to the system at left that include forces associated with 
motion, pen-tip forces due to writing, and reflected light for optical read int. 
These physical inputs are applied respectively to position sensor 210. 
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P'«su,. s.nso,. ,00. „p,ica, sensors ^30. jeo. T^e senso, 

(MUX, 275. MUX 275 ma, ^ eft.e, a «me*isi„„ „e,„encv 

m«p,„e,. Overa..co„,ro,oMheP,s,s,e..p.o..e.5,con,ro„e,270 

^ Base tinij q o 

8as» „n« 92. shown m figure ,2. communica,es vvim P, 9, 
'ece,Vi„9 sensor signals, processes ,ne sensor signals, an. drives an, 

nolebook Keyboa« oompoter con,pnsing a CPU 940. keyboard 943 
■ --orv 942. and display 95. ,„ addllion. ine signal condllloning on,', 9S 
-eludes a iransmmer/receiver as pan o. Ihe communication cnanne, 
.n.-ace 9S, con,mu„ica,l„„s link 93. and de™« 962. and signal 
-ndlLonlng elecronics ,9S3.9es, lor precond«ioning sensor signals Iron, 
—ace 9S, and analog-lcd^iia, conveders ,o, ,ee«„g digi,.ed senso, 
-■aled s^nals inrougn ™, 957 ,o ,ne CPU .or processing. Tne base un« 
can also conned ..ih Wide Area Ne,»orks Ihrougn VO interlace 944 and is 
able ,0 relneve and modiV eleclronic versrons o, ,he ose.s documenis An 
op onal auxiliary processor 94, (snown dolled, can be used ,o augmen, 

340 lor execuling wrillen message recognilion algorilhms lha, require 
-ore .niensive digila, signal p,ocess.ng man ina, usual,, done by Ihe CPU 

Advances in mico-elearonlc lechniques can allo„ mos, ol Ihe 
-uoc«ons o, ,ne base uni, ,„ be incorpora,ed ,n ihe PI unii, paniculan, ine 
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sensor signal conditioning, binary coded signal generation, and cache 
memory for buffering and temporary storage. 
^- Other Annlicalionc! an^ Variatip r^ff 

The above description of a CB system that uses a pen-instrument 
and base system was used as a convenient means to describe the 
operation of a manual entry interactive paper and electronic document 
handling and processing system. 

It should be noted that had any of the other applications listed in 
section A of the Detailed Description, the basic system configuration would 
be the same: 

(a) a physical document with a writing surface and pre-printed 
inlormation; 

(b) a pen-instrument for reading pre-printed (bar code) 
information, for sensing writing forces, for sensing 
pen-instrument motion, for sensing written characters, and for 
transmitting the information to a base unit; and 

(c) a base unit for receiving information generated by the 
pen-instrument, interpreting the information and modifying a 
stored electronic version of the physical document and 
displaying the interpreted results to the user for acceptance or 
correction. 

Therefore, the CB system description has served as an adequate 
vehicle for supporting a general manual entry interact.ve paper and 
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electronic document handling and processing system. A specialized 
application program package would customize the system described (or 
each distinct application and document type. 

Although the previous description of the PI and its use emphasized 
optical reading and message writing on a document surface, it should be 
recognized that the PI may. also provide information from its gyroscopes 
and/or accelerometers for describing non-contact gestures unassociated 
with physical writing on a document. Two cited examples of this form of 
input to the system are the "check" signal indicating affirmation (correct) 
and the "X" signal indicating negation (wrong). Other gestures may be 
designed for use in a given context to indicate special messages (e.g., in 
editing, horizontal oscillatory motions may indicate deletion if performed 
over a previously written message or it may indicate an "underline" if not 
performed over a written message but under the message). 

Selective use of color may also be used to distinguish entries from 
authorized multiple users by generating an I.D, code for each users PI at 
the outset of writing session. A color display could readily identify the 
source of entries by using a different color coding for each user. 

One or more buttons may be provided to initiate certain actions such 
as indicating that a bar code is to be read. 

The PI can be used for other purposes than writing. For example, 
the PI can function as a pointer or as a "master mouse" that drives a slave 
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mouse cursor on a graphic display or lor navigating 3-dimensional vinual 
reality displays. A button can be used as a mouse button. 

Yet another variation of the PI can have a receiver lor receiving the 
output of multiple (infrared) transniitier for position location using 
5 triangulation techniques. This variation could be used for blackboard 
writing or other large surface writing and display boards. 

In the foregoing specification, the invention has been described with 
reference to specific embodiments thereof. It will, however, be evident that 
various modifications and changes (as described above) may be made 
10 thereto without depanmg from the broader spirit and scope of the invention 
as set forth in the appended claims: The specification and drawings are. 
accordingly, to be regarded in an illustrative rather than a restrictive sense. 



CLAIMS 
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1 • A manual entry data handling system for storing data 
representative of motion of a „,anual instrument ir 



n a 



multi-dimensional space, the system comprising: 

(a) a first communication channel; 

(b) a manual instrument having at least one transducer for 
sensing self motion in the muifi-dimensional space, generating a 

Signal representative of the self motion, and a first communication channel 
interface for accepting and sending the signal over the first 

communication channel; and 

(c) a base processing unit with a control processing unit. (CPU), a 
first communication channel interface coupled to the first communication 
Channel for accepting the signal sent by the manual instrument 

over the communication channel and for storing a representation of the 

signal into a memory under control of the CPU so that the stored 
representation can be retrieved for reconstructing the self motion of the 
manual instrument. 

2. The system of claim i wherein the base processing unit is 
contained within the manual instrument and further comprises a second 
communication channel interface for coupling the CPU to an external bus 
for control and data communications to other devices. 



3. The system of claim 1 wherein the manual instrument 
transducers for sensing self motion are rate gyroscopes thai generate a set 
of electrical signals, each electrical signal representative of a component of 
motion in the multi-dimensional space. 

4. The system of claim 1 wherein the manual instrument 
transducers for sensing self motion are accelerometers that generate a set 
of electrical signals, each electrical signal representative of a component of 
motion in the multi-dimensional space. 

5. The system of claim l wherein the manual instrument 
transducers for sensing self motion are a combination of at least one rate 
gyroscope and at least one acceleromeier for generating a set of electrical 
signals, each electrical signal representative of a component of motion in 
the multi-dimensional space. 

6 The system of claim 1 further comprising at least two infrared 
receiving scanning stations located at prescribed positions external to the 
manual instrument for scanning the area in which the manual instrument is 
located, and wherein the manual instrument transducer for sensing self 
motion is an infrared transmitter for transmitting an infrared signal that is 
detected by the infrared receiving stations for determining the manual 
instrument position by trianguJation techniques. 

7. The system of claim 1 further comprising at least two infrared 
radiating scanning sources located at prescribed positions external to the 
manual instrument for illummating the manual instrument, and wherein the 
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manual instrument transducer for sensing self motion is an infrared detector 
for detecting the illumination by each external infrared source for 
determining the manual instrument position by triangulation techniques. 

8. The system of claim 1 wherein the manual instrument is for use 
as a writing instrument. 

9. The system of claim 1 wherein the manual instrument is for use 
as a pointer, 

10. A manual entry interactive document handling system using a 
writing instrument to write data on a document and interpret 
the written data so that a corresponding electronic document is modified by 
incorporating the written data, the electronic document data suitably 
encoded as binary data, the system comprising: 

(a) a physical document page with a surface suitable for writing 

upon: 

(b) a communications channel; 

(c) a pen-instrument having a writing point for writing data on the 
physical document page using a visible writing medium, including a set of 
transducers for sensing the written data by generating electrical signals 
representative of physical actions exerted or the pen-instrument when used 
for writing, and a communications interface for sending the electrical signals 
over the communications channel; and 

(d) a base processing unit including: 
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(i) a communications interface for interfacing with the 
communications channel and receiving the electrical signals from the 
pen-instrument, 

(ii) a signal conditioning unit for converting the electrical 
signals into binary coded data; 

(iii) a central processing unit (CPU) with a memory for 
programmable control of the base processing unit for storing and modifying 
documents stored in the memory in conformance with the electrical signals 
received from the pen-instrument. 

1 1. The system of claim 10, wherein the physical document page 
has pre-printed data in support of an application for which the physical 
document page is intended and a corresponding image stored in the base 
processing unit. 

1 2. The system of claim 1 1 . wherein the physical document page 
has a visual binary encoded signal for identifying the application, the visual 
binary encoded signal suitable for reading using an optical sensor located 
in the pen-instrument. 

13. The system of claim 12. wherein the physical document page 
is a calendar page and the visual binary encoded signal identifies a 
calendar segme^nt as.the nftifin dai_segment represented- by the physical 
document page. 

14. The system of claim 13. wherein the document page has a 
second visual binary encoded signal suitable for reading using an optical 
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sensor located in the pen-instrument for identifying a location of a 
prescribed application area on the physical document page. 

1 5. The system of claim 14 wherein the second visual binary - 
encoded signal identifies the location of a prescribed application as a 
calendar segment that is a fraction of the physical document page calendar 
segment. 

1 6. The system of claim 10 wherein the physical document page 
has a visual binary encoded signal for identifying a user associated with the 
physical document page. 

1 7. The system of claim 10 wherein the pen-instrument visible 
writing medium is ink. 

1 8. The system of claim 10 wherein the pen-instrument visible 
writing medium is pencil-graphite. 

1 9- The system of claim 10 wherein the pen-instrument visible 
writing medium is chalk. 

20. The system of claim 1 0 wherein the pen-instrument further 
includes an optical sensing device for reading a visual binary encoded 
Signal imprinted on the physical document page. 

21 - The system of claim 20 wherein the optical sensing device is a 
charge coupled device (CCD). 

22. The system of claim i o wherein the pen-instrument set of 
transducers is a sei of accelerometers. 
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23. The system of claim 1 0 wherein the pen-instrument set of 
transducers is a set of rate gyroscopes. 

24. The system of claim 1 0 wherein the pen-instrument set of 
transducers includes at least one rate gyroscope and at least one 
accelerometer. 

25. The system of claim l o wherein the pen-instru.ment further 
comprises a pressure transducer for generating an electrical signal 
representative of the force applied to the pen-instrument writing point when 
writing on the physical document page surface. 

26. The system of claim 25 wherein the pressure transducer is a 
set of strain gauges arranged to generate a set of electrical signals 
representative of three orthogonal components of force applied to the 
pen-instrument writing point with two-onhogonal components in the physical 
document page surface and a third perpendicular to the same surface. 

27. The system of claim 1 0 wherein the pen-instrument further 
comprises an optical array sensor located near the pen-instrument writing 
point for reading a local optical image in the vicinity of the pen-instrument 
writing point. 

28. The system of claim 27 wherein the optical array sensor is a 
charge coupled device array. 

29. The system of claim 1 0 wherein the communication channel is 
a radio link. 
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30. The system of claim 10 v^herein ,he communication channel is 
an infrared channel. 

3t. A melhof io, ,n,„ual emy o. «.a Imo a documen, haratno 
s/.,em u«„9 a p.„.i„s,rum.n, having a wming po,n, ,o, ^.-.n, aa,a on a 
Physical ao«,„en, an. 

ccresponaing eleconic doco^n, is n,«i,iea by inco,po,a«ng ,He «„en 
aa«. ,ne pnysica, doco^n, Having a prescribed p.„,ed ,„,™, , 
ba, code ,0, do«.„e« iden-iiicaiion. ,ne documen, Handiing s,s,en, having 
a s,ored etoronic copy c, ,he physioa, documem. ,he pen.ins,„,™n, 
havng „,eans ,o, deiecing a se, ol physical forces acing on ,he 
P-ins„un,en, when „n,ing. ,he se, o, physica„c,ces iordescnoing ,he 
»r,„en da,a on ,he physical documen, including ,he de,ec,icn o, ,o,ces 
acng cn ,he pen.ins,n,men, „,i,i„g and means .or reading ,Ke 

documen, idemilicaiion ba, code. ,he mchod comprising: 
(al reading ihe bar code using ,he pen-inslrumen,: 

(bl -""""S . he pe,.,ns,rume„,,o an ini,ial,oca,ion on, he physical 
aocumen, a, wh^ „n„„g ,s ,o begin as indicaied by .orces acing o„ ,he 
pen-mstrument writing poin-; 

(c) computing the .nitia. locat.on from the forces required to move 
.he pen-insio^ent from tne bar code location .0 the inftial location- 

(^) writing on the physical document using the pen-instrument 
wnting point and detecting a set of forces acting on the pen-instrumen, while 

writing; 
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(e) reconstructing the written data electronically from the detected 
set of forces; 

(0 superimposing the reconstructed written data on to the stored 
electronic copy of the physical document. 

32. The method of claim 31 wherein the reconstructed written data 
is representative of the writing point locus traversed on the physical 
document during writing. 

33. The method of claim 31 wherein the reconstructed written data 
is an alphanumeric interpretation of the written data. 
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